Index server architecture using tiered and sharded phrase posting lists

ABSTRACT

An information retrieval system uses phrases to index, retrieve, organize and describe documents. Phrases are extracted from the document collection. Documents are the indexed according to their included phrases, using phrase posting lists. The phrase posting lists are stored in an cluster of index servers. The phrase posting lists can be tiered into groups, and sharded into partitions. Phrases in a query are identified based on possible phrasifications. A query schedule based on the phrases is created from the phrases, and then optimized to reduce query processing and communication costs. The execution of the query schedule is managed to further reduce or eliminate query processing operations at various ones of the index servers.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/694,780, filed on Mar. 30, 2007, entitled “INDEX SERVER ARCHITECTUREUSING TIERED AND SHARDED PHRASE POSTING LISTS,” now U.S. Pat. No. ______and is related to the following co-pending applications: QUERYSCHEDULING USING HIERARCHICAL TIERS OF INDEX SERVERS, filed Mar. 30,2007; INDEX UPDATING USING SEGMENT SWAPPING, filed Mar. 30, 2007; PHRASEEXTRACTION USING SUBPHRASE SCORING, filed Mar. 30, 2007; and BIFURCATEDDOCUMENT RELEVANCE SCORING, filed Mar. 30, 2007, all of which areco-owned, and incorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates to information retrieval systems for largescale document collections, such as the Internet.

BACKGROUND OF THE INVENTION

Information retrieval systems, generally called search engines, are nowan essential tool for finding information in large scale, diverse, andgrowing information systems such as the Internet. Generally, searchengines create an index that relates documents (or “pages”) to theindividual words present in each document. The index is typically storedas an inverted index, in which for each unique term in the corpus, thereis stored a posting list identifying the documents that contain theword.

A document is retrieved in response to a query containing a number ofquery terms, typically based on having some number of query termspresent in the document. Very generally, this is done by decomposing thequery into its individual terms, and the accessing the respectiveposting lists of the individual terms. The retrieved documents are thenranked according to other statistical measures, such as frequency ofoccurrence of the query terms, host domain, link analysis, and the like.The retrieved documents are then presented to the user, typically intheir ranked order, and without any further grouping or imposedhierarchy. In some cases, a selected portion of a text of a document ispresented to provide the user with a glimpse of the document's content.

Direct “Boolean” matching of query terms has well known limitations, andin particular does not identify documents that do not have the queryterms, but have related words. For example, in a typical Boolean system,a search on “Australian Shepherds” would not return documents aboutother herding dogs such as Border Collies that do not have the exactquery terms. Rather, such a system is likely to also retrieve and highlyrank documents that are about Australia (and have nothing to do withdogs), and documents about “shepherds” generally.

The problem here is that conventional systems index documents are basedon individual terms, rather than on concepts. Concepts are oftenexpressed in phrases, such as “dark matter,” “President of the UnitedStates,” or idioms like “under the weather” or “dime a dozen”. At best,some prior systems will index documents with respect to a predeterminedand very limited set of ‘known’ phrases, which are typically selected bya human operator. Indexing of phrases is typically avoided because ofthe perceived computational and memory requirements to identify allpossible phrases of say three, four, or five or more words. For example,on the assumption that any five words could constitute a phrase, andthat a large corpus would have at least 200,000 unique terms, therewould be approximately 3.2×10²⁶ possible phrases, clearly more than anyexisting system could store or otherwise programmatically manipulate. Afurther problem is that phrases continually enter and leave the lexiconin terms of their usage, much more frequently than new individual wordsare invented. New phrases are always being generated, from sources suchtechnology, arts, world events, and law. Other phrases will decline inusage over time.

Some existing information retrieval systems attempt to provide retrievalof concepts by using co-occurrence patterns of individual words. Inthese systems a search on one word, such as “President” will alsoretrieve documents that have other words that frequently appear with“President”, such as “White” and “House.” While this approach mayproduce search results having documents that are conceptually related atthe level of individual words, it does not typically capture topicalrelationships that inhere between co-occurring phrases themselves.

Another problem with existing individual term based indexing systemslies in the arrangement of the server computers used to access theindex. In a conventional indexing system for large scale corpora likethe Internet, the index comprises the posting lists for upwards of200,000 unique terms. Each term posting list can have hundreds,thousands, and not infrequently, millions of documents. The index istypically divided amongst a large number of index servers, in which eachindex server will contain an index that includes all of the uniqueterms, and for each of these terms, some portion of the posting list. Atypical indexing system like this may have upwards of 1,000 indexservers in this arrangement.

When a given query with some number of terms is processed then in suchan indexing system, it becomes necessary to access all of the indexservers for each query. Thus, even a simple single word query requireseach of the index servers (e.g., 1,000 servers) to determine whether itcontains documents containing the word. Because all of the index serversmust process the query, the overall query processing time is limited bythe slowest index server.

SUMMARY OF THE INVENTION

An information retrieval system and methodology uses phrases to indexand search documents in the document collection. Phrases are also usedto decompose inputs (e.g., queries) into phrase trees, to schedule andoptimize the execution of searches, to support the updating andmaintenance of the indexes, and to support bifurcated relevance scoringof documents.

In one aspect, an information retrieval system includes an indexingsystem and index server architecture based on phrases. Phrases areextracted from a document collection in a manner that identifies realphrases as used in by language users, as opposed to mere combinations ofwords. Generally, this is done by collecting a large body of wordsequences that are candidates phrases based on the structural featuresin the documents. Each candidate phrase is given a document phrase scorefor each document in which it appears, in a manner that reflects itslikelihood of being a real phrase based on its position within adocument, and the extent to which it occurs independently or jointlywith other candidate phrases in the document. In addition, eachcandidate phrase is processed so as to identify any subphrases therein,which are similarly scored. Each candidate phrase's individual documentphrase scores are then combined across the documents in which it appearsto create a combined score. The document phrase scores and the combinedscore for a candidate phrase are evaluated to determine how strongly thedocument collection supports the usage of the candidate phrase as a realphrase. Generally, a candidate phrase is retained where it is stronglysupported by at least one document; for example the maximum of itsindividual document phrase scores exceeds a predetermined threshold. Acandidate phrase is also retained where it is moderately supported, asindicated by having a combined phrase score above a second predeterminedthreshold. This shows that the candidate phrase has a sufficientwidespread use to be considered a real phrase. Finally, a candidatephrase is also retained where it is broadly supported, as indicated bythe phrase receiving a minimum score from some number of documents. Asan example, the system can include approximately 100,000 to 200,000phrases, which will represent real phrases used in documents, ratherthan mere combinations of words.

Given a set of phrases (whether assembled in the foregoing manner or byother methods), the documents can be indexed by the phrase. For eachphrase, a phrase posting list is maintained that identifies documentsthat are associated with the phrase; a document can be associated with aphrase by containing at least one occurrence of the phrase or anoccurrence of a related phrase (e.g., a synonym, a subphrase, etc.). Agiven phrase posting list can have any number of documents, includingmillions of documents for extremely common phrases. These phrase postinglists are assigned for serving to plurality of index servers. The phraseposting list assignments are made so as to minimize the inter-servercommunications required for subsequent query processing. There arevarious embodiments of how the phrase posting lists can be assigned tothe index servers, which have the following general structure.

One aspect of the indexing system is to divide each of the phraseposting lists into a number of partitions called “shards.” Within agiven phrase posting list there are identified the documents associatedwith the phrase. These documents are then assigned across some set ofshards using a shard assignment function, which has the property that agiven document is always assigned to the same shard, regardless of thephrase posting list in which it appears. Multiple shard assignmentfunctions can be used, so that different shard assignment functions canbe used with different phrase posting lists. The shard assignmentfunction for a given phrase posting list is selected based on attributesof the phrase and its posting list. Each of the shards is then assignedto one of the index servers. The shard will be stored at the indexserver, either in memory or on disk or a combination of both. Typicallyeach index server will store the shards for one or more than one phraseposting lists. In addition, a given shard may be stored in duplicate ata number of index servers, so as to provide increased query processingcapability.

Another aspect of the storage arrangement for the phrase posting listsis the use of tiers. In this aspect, each phrase posting list isassigned to one of a number of tiers. The phrase posting lists in eachtier will be stored in one or more index servers associated therewith.Thus, a number M of tiers are established, each tier associated with aset of index servers. For example, a system can be arranged with first,second and third tiers, though additional tiers can be used. Thus, allphrase posting lists assigned to the first tier are stored in the firsttier index servers, and so on for the remaining tiers of index servers.Phrases are assigned to tiers using a phrase assignment function thatrepresents the query processing costs for a given phrase posting list,and assigns phrases of similar cost to the same tier.

The use of shards can take advantage of the tiers of the index servers.In one embodiment, each tier is associated with a number of shards,which can be selected based on the performance demands and capacities ofthe index servers in the tier. A shard assignment function is then usedfor each tier to partition the phrase posting list assigned to the tieramongst the shards. As before, each shard is stored in one of the indexservers associated with that tier. The shard assignment function is suchthat a given document is always assigned to the same shard. Therelationship of the tiers to each other with respect to the number ofshards therein can also be beneficially selected to further improvequery processing. One way to do this is to associate each tier with anumber of shards S, such that S for the M^(th) tier is an integermultiple k of S for the (M-1)^(st) tier. Thus, the number of shardsincreases from the first tier to the last tier. This partitioning of thephrase posting lists into shards ensures that any given index serverneed only communicate with a restricted set of index servers in the nexttier in order to perform a query intersection. As a result, the overalltime for query processing is significantly reduced. This is in contrastto conventional indexing systems which require any index server to beable to communicate with any other index server during query processing.

Another aspect of the information retrieval system is a method formaintaining and updating the index servers, so as to ensure generallycontinuous and availability of the index. In this aspect, the index isdivided into a number of segments, which are independent of the tiersand shards. A segment may store the index information for any number ofdocuments. The phrase posting lists in each segment can then be tiered,sharded and assigned to the index servers as described, so as to formsegment shards. A given index shard is thus associated with the set ofcorresponding segment shards. For example, if there are twenty segmentsin the index, then a given index shard for group of phrase posting listsin a designated tier and shard is associated with the correspondingtwenty segments shard for this same group, tier and shard of phraseposting lists. To update an index shard then, only the most recentlyupdated segment shards need be merged into a copy of the index shard.This merging process can be handled by each index server for itsrespective shards, or by a separate set of merging servers. Once theindex shards are merged, they are swapped in to replace the existingversions being served by the index servers. The swapping is done so thateach index server remains available for query processing for most of thephrases it in the shards it stores, while the newly updated shards arebeing stored to index server.

Another aspect of the information retrieval system is a method ofphrasification, which is identifying a set of phrases within any inputtext comprising a set of terms (and optionally operators), such as inputsearch queries. An input text is accepted in the form of a Boolean treeof words and operators (either express or inferred). The tree isflattened to form a disjunction of conjuncts, where each conjunctcontains a subset of words of the input. The words in each conjunct arethen processed to generate a set of possible partitionings of the wordsinto phrases. These partitionings are called “phrasifications.” Forexample, the given the set for words “New” “York” “restaurants”, thepossible phrasifications are:

“New” “York” “restaurants”;

“New York” and “restaurants”;

“New” and “York restaurants”; and

“New York restaurants”.

Each phrasification is scored using a scoring model that incorporatesthe expected probability of the phrase occurring in a document, thenumber of phrases in the phrasification, a confidence measure of eachphrase, as well as adjustment parameters for controlling the precisionand recall of searches on the phrases. Some number of highest scoringphrasifications are then selected as best representing the phrases inthe input text.

Another aspect of the information retrieval system is a method of queryscheduling the creates an schedule for executing a search of the phrasesof a query so as to minimize query processing costs by the indexservers, as well as to reduce inter-server communications. Generally,the input phrases are presented in a phrase tree that represents thelogical structure of phrases in the query using a set of nodes for thephrases and operators. The phrase tree is processed with a set ofscheduling rules that assign each phrase only to the index servers whichserve the shard containing the phrase. Thus, instead of having everyindex server process a phrase, only the index servers that contain thedocument in their respective shards will do so. The scheduling rulesalso assign a query cost to the nodes that represents a cost forprocessing the nodes. These costs are then used to initially sequencethe order of execution of the nodes in the phrase tree so as to reduceinter-machine communication. The scheduling rules also assign to eachoperator node in the query tree a set of index servers that will beresponsible for evaluating it; these assignments are made so as tominimize the amount of inter-server communication.

Once the phrase tree is initially scheduled, it can be optimized furtherby analysis of the assigned costs at each node using some cost function.Particular nodes are identified in the phrase tree that can betransformed into their logical equivalents. A cost function is appliedto the existing node and its logical equivalent, and the form having thelower cost is selected for inclusion in the optimized phrase tree. Thecost functions can be based on the tier assignments of each phrase, thelength of each phrase, or a cost of a subtree of the node. An optimizedphrase tree is then used as a query schedule by the index servers. Thescheduling and optimization, in conjunction with the tiers and shardingensures that each index server will process only those portions of thephrase tree which can be affected by its phrase posting lists.

Another aspect of the information retrieval system is a method by whicha query schedule is executed by the set of tiered and sharded indexservers. The query schedule identifies for each node which index serverthe node is assigned to. A recursive descent through the phrase treethen is used, and at each node, the assigned index server determineswhether any of the children node of the current node are at the indexserver, or at another index server. Local children nodes—i.e., phrasesmat are stored at the index server—are processed to create a list ofdocument. The index server then transmits this document list either backto the index server that called it, or to a next index server forfurther processing. Again, the index servers can use the shardinformation to ensure that these transmissions are optimized in length(smallest possible lists) and to only the index servers that can makeuse of the list.

In one embodiment of the information retrieval system, a bifurcateddocument relevance scoring model is provided. Here, the final documentscore for each document in the search results is based on two scoringfunctions. A first scoring function, a phrase relevance scoringfunction, is applied during document indexing. This function provides ascore of the relevance of a given phrase to a given document,independently of any query or other phrases. A second scoring functionis used during query processing. This second scoring function takes asinput the phrase relevance scores for a set of documents, and thephrases of a query, and determines a final relevance score for eachdocument. This bifurcated scoring model enables for improved queryprocessing efficiency, as well as reduced storage requirements for thephrase posting lists.

The present invention has further embodiments in computer system andsoftware architectures, computer program products and computerimplemented methods, and computer generated user interfaces and searchresult contents.

The foregoing are just some of the features of an information retrievalsystem and methodology based on phrases. Those of skill in the art ofinformation retrieval will appreciate the flexibility of generality ofthe phrase information allows for a large variety of uses andapplications in indexing, searching, ranking, a document informationprocessing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of the software architecture of one embodimentof the present invention.

FIG. 2 is a block diagram of the indexing system.

FIG. 3 is a flow diagram of a method for phrase extraction.

FIG. 4 is a block diagram of one embodiment of the storage architecturefor the phrase posting lists in the index servers.

FIG. 5 is a block diagram of another embodiment of the storagearchitecture for the phrase posting lists in the index servers.

FIG. 6 is a block diagram of yet another embodiment of the storagearchitecture for the phrase posting lists in the index servers.

FIG. 7 is an illustration of the merging of segment shard files intoindex shard files.

FIG. 8 is an illustration of the query phrasification, scheduling andexecution processes in the front end server.

FIG. 9 is an illustration of the query phrasification process.

FIG. 10 is an illustration of the query scheduling process.

FIGS. 11A and FIG. 11B illustrate optimization by factoring of commonnodes in a query schedule.

FIGS. 12A and FIG. 12B illustrate an example of query execution on oneor more index servers.

The figures depict a preferred embodiment of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION OF THE INVENTION

I. System Overview

Referring now to FIG. 1, there is shown the software architecture of aninformation retrieval system 100 in accordance with one embodiment ofpresent invention. This embodiment of the information retrieval system100 includes an indexing system 110, a search system 120, a userinterface server 130, a set of index shard files 115, and a documentinformation database 155. The search system 120 includes a front endserver 140, a document information server 150, and an index servercluster 160 comprising a plurality of index servers 200.

The indexing system 110 identifies phrases in documents, and indexes thedocuments according to their phrases, by accessing various websites 190and other document collections over the network 180, and processingdocuments contained therein. The indexing system 110 maintains the indexinformation in the index shard files 115. The indexing system 110 alsoupdates the document information database 155 with content pertaining toindexed documents, including cached document versions, metadata, andrelated document information. The user interface server 130 receives aquery from a client 170, and passes the query, in the form of a Booleanquery word tree to the front end server 140. The client 170 can be abrowser (for example to allow individual users to submit queries), or itcan be any application program with programmatic access to anapplication programming interface exposed by the user interface server130 (e.g., a separate server application that queries the system 100 fordocuments). Additionally, the front end server 140 can also expose anapplication programming interface that allows other applications todirectly input properly formed queries.

The front end server 140 decomposes the query tree into a phrase treethat is optimized for execution by the index servers 200. The front endserver 140 provides the optimized phrase tree to the search system 120.The search system 120 is responsible for managing execution of thesearch by the index servers 200. The index servers 200 process thephrase tree by accessing the index shard files 115, and returning searchresults comprising a set of documents relevant to the phrases of thequery. The front end server 140 receives the search result set, and thenranks the documents in the search results, as well as accessing thedocument information server 150 to obtain document level information.The search system 120 provides the search results and associateddocument information to the user interface server 130. The userinterface server 130 formats the search results, including creating thelayout of the search results and associated document information, andother supplemental information or content (e.g., advertisements), andproviding the formatted search results back to the client 170.

In the context of this system 100, “documents” are understood to be anytype of media that can be indexed and retrieved by a search engine,including web documents, images, multimedia files (e.g., audio, video),text documents, PDFs or other image formatted files, and so forth. Adocument may have one or more pages, partitions, sections or othercomponents, as appropriate to its content and type. Equivalently adocument may be referred to as a “page,” as commonly used to refer todocuments on the Internet. No limitation as to the scope of theinvention is implied by the use of the generic term “documents.” Thesystem 100 operates over a large corpus of documents, such as documentsvariously collected from across the Internet and World Wide Web, but canlikewise be used in more limited collections, such as for the documentcollections of a library or a private enterprise. In either context, itwill be appreciated that the documents are typically distributed acrossmany different computer systems and sites. Without loss of generalitythen, a set of documents generally, regardless of format or location(e.g., which website or database) will be collectively referred to as acorpus or document collection. Each document has at least one associatedidentifier that uniquely identifies the document; these identifiers canbe URLs, as well as document numbers, hashes, etc.

II. Indexing System

The indexing system 110 provides three primary functional operations: 1)identification of phrases 2) indexing of documents with respect tophrases, and 3) generation and maintenance of the phrase-based indices.Those of skill in the art will appreciate that the indexing system 110will perform other functions as well in support of conventional indexingoperations, and thus these other operations are not further describedherein.

Referring to FIG. 2, there is shown in the indexing system 110 infurther detail for one embodiment. The indexing system 110 includes aserver cluster master 220, a set of segment shard files 225, a swapmaster server 240, and a phrase identification server 250, and adatabase of phrase data 255. The indexing system 110 communicates withthe index server cluster 160, the document information database 155, andthe index shard files 115. The operations and responsibilities of thevarious servers will be further described below. Each of the variousservers is implemented as server program executing on server-classcomputer comprising a CPU, memory, network interface, peripheralinterfaces, and other well known components. The computers themselvespreferably run an open-source operating system such as LINUX, havegenerally high performance CPUs, 1 G or more of memory, and 100 G ormore of disk storage. Of course, other types of computers can be used,and it is expected that as more powerful computers are developed in thefuture, they can be configured in accordance with the teachings here.

1. Phrase Extraction

The first function of the indexing system 110 is to identify validphrases for use in indexing documents; this process is called phraseextraction, since it seeks to extract valid (or “real”) phrases from thedocument collection. This functionality is principally performed by thephrase identification server 250. FIG. 3 illustrates an overall flowdiagram of the phrase extraction process. There are four basic stagesthe phrase identification process:

302: Extract initial set of candidate phrases;

304: Identify valid phrases based on document collection information;

306: Remove redundant phrases;

308: Refine the phrase list with heuristics.

Each of these stages will now be discussed in further detail.

(1) Initial Phrase Extraction

The phrase identification server 250 generates 302 an initial list ofphrases by iterating over each document in a set of documents,identifying candidate phrases within each document, and then testing thecollected candidates to determine the whether they are valid. The set ofdocuments for this initial process can be the entire documentcollection, or a subset thereof, such as set of the most recentlycrawled documents; the number of documents is preferably on the order of2 to 5 billion documents.

The phrase identification server 250 scans over each document's body andanchors, and maintains a buffer of the last N words of text encountered;N is preferably between 5 and 20 words. A “hit” is an event during thisiteration over the documents where the phrase identification server 250adds a candidate phrase instance to a phrase map table stored in thephrase data 255. A hit can arise when the buffer is full, or when thephrase identification server 250 identifies semantically bounded textsequences, here called “text breaks.” A semantically bounded textsequence is identified by a semantic boundary marker that indicates thatthe next word to be read is semantically disjoint from the previous wordor set of words in the buffer. These markers include, for example, asentence boundary, a paragraph boundary, a positional boundary (such asthe end of the title), a visible HTML element such as a table orhorizontal rule, or a significant change in typeface (e.g., change infont style, size).

Whenever the phrase identification server 250 encounters a text break,or whenever the buffer contains N words, the phrase identificationserver 250 adds the contents of the buffer (forming a word sequence) toa phrase map table. If a text break is found, then the buffer iscleared; if the buffer is full, then the first word is dropped from thebuffer, and the next word in the text is read in. If a hit isencountered that would result in a single word being written out as atext break, than that hit is ignored, and the phrase identificationserver 250 continues to the next word in the document.

The phrase map table maps each received word sequence to informationabout the visual position of the word sequence in the document, such aslocation within document, and the flag indicating position of the end ofthe word sequence relative to the start of the buffer, as well astypeface characteristics (such as font, size, and style), and length.The location can be indicated by section indicators (e.g. title, body,list, table) or word position. Hence, the phrase map table has thefollowing logical structure:

Phrase Map Table: <word sequence, location, hit position, length,typeface>.

There are four possible values for the hit position: Initial, Final,Exact, Medial.

An Initial hit flag indicates that this particular sequence of N wordswas the first sequence of N words after the buffer was emptied.

A Final hit flag indicates that the text break was the last sequence ofN words prior to an emptying.

An Exact hit flag indicates that the text break satisfies both of theabove criteria.

A Medial hit flag indicates that the text break does not satisfy eitherof the above criteria.

Additional information can also be recorded for each word sequence suchas linguistic markers (e.g., capitalization), or inclusion of the phraseas anchor text in a hyperlink. The order of the elements in the phrasemap table is not limited to that shown above.

If there are multiple hits on the same sequence of words, the phraseidentification server 250 keeps all of this information in the phrasemap table. The phrase identification server 250 also may applylinguistic rules to the text break to further categorize it. Forexample, if a sequence of words appears to be a proper noun (such as asequence of capitalized words in English text), that can be indicated inthe phrase map table as well.

The following is an example of the buildup of the phrase map table for asimple HTML document:

<title>The quick brown fox</title>

The quick brown fox jumps over the lazy god.

Here, the phrase identification server 250 uses N=5. As the phraseidentification server 250 scans through these words one at a time, thefollowing occurs. An simple version of the phrase map table is includedhere, with only the word sequence, location, and hit position flagsshown:

Example 1

Phrase Map Table (Word sequence, Word location, position, Parsed BufferContents length) Comment <title> -empty- <title> indicates text break,but buffer is empty so no hit occurs. The The quick The quick brown Thequick brown fox The quick brown fox </title> -empty- “The quick brown</title> indicates text fox”, title, “exact”, 4 break. Buffer cleared.The The quick The quick brown The quick brown fox The quick brown foxjumps The quick brown fox jumps “The quick brown fox Buffer full.jumps”, body, initial, 5 over quick brown fox jumps over “quick brownfox Buffer full. jumps over”, body, medial, 5 The brown fox jumps overthe “brown fox jumps Buffer full over the”, body, medial, 5 lazy foxjumps over the lazy “fox jumps over the Buffer full lazy”, body, medial,5 god jumps over the lazy god “jumps over the lazy Buffer full. god”,body, medial, 5 . -empty- “over the lazy god”, “.” indicates text body,final, 4 break.

After processing the document, the phrase map table contains a set ofword sequences, each a string with between two and N words, along withadditional information about the word sequence.

For each word sequence, the phrase identification server 250 assigns aphrase score that indicates how good the word sequence is as a candidatephrase. The scoring of the word sequences can be done after eachdocument is processed, after some set of documents is processed, or eachtime a word sequence is added to the phrase map table. The scoringfunction provides a scaled score (e.g., 0-100) that takes intoconsideration the location of each word sequence and its position, andoptionally its typeface characteristics. In one embodiment, wordsequences that are “exact” position hits are most highly scored,followed by “initial” hits, “medial hits”, and then “final” hits. In analternative embodiment, the initial, medial and final hits are givenequal scores, and the exact hit is given a higher score. The score isalso scaled by the location of the hit, with locations towards the startof the document (e.g., title) being more highly scaled. Typeface canalso be used for scaling, including increasing the score for bold text,or a large font size. Use of the word sequence as a hyperlink anchoralso increases its score; the upweight can be based on some measure ofthe quality of the linked page, such as its PageRank.

Generally then, the phrase identification server 250 identifies a wordsequence as a good candidate phrase if its score is above an initialphrase identification threshold. This threshold may be a function of thenumber of words in the sequence, so that the threshold value is lowerfor longer sequences, since very meaningful long phrases occur morerarely than short ones. In general, the scoring function for the initialthreshold is constructed so that even a single occurrence of a propernoun or text in a highly emphasized position like the title is enough topass the initial threshold. If the hit had a score above the threshold,the word sequence is added to a candidate phrase table along with itsscore.

Next, the phrase identification server 250 determines whether or notsubphrases of each candidate phrase are of interest as potentialcandidate phrases themselves. For instance, in the example above “Thequick brown fox jumps” is not likely a candidate phrase, but “The quickbrown fox” alone is—and the phrase identification server 250 does notdiscard the information that this sequence occurred in the body as wellas the title.

More generally then, to identify subphrases that are potential candidatephrases, the phrase identification server 250 devolves the score of anycandidate phrase “A B . . . C D” (with up to j=N words) by decomposingthe candidate phrase into two child subphrases “A B . . . C” and “B . .. C D” (with j-1 words each), and scoring each of these subphrases. Thescoring rules applied to these subphrases depend on whether thecandidate phrase is a strong phrase, as described below. If j is 2, thenthe phrase identification server 250 skips this step.

The devolution rule is designed to avoid double counting, and so itdepends on the position of the hits. First, the phrase identificationserver 250 compares the phrase's score to a second threshold, which isused to further identify strong phrases. If a candidate phrase has ascore above the strong phrase threshold, it is determined to be likelyto be a real phrase, and thus the phrase's score need not be furtherdevolved to its subphrases. For example, if the phrase “new york giants”appears has a score above the strong phrase threshold, the phraseidentification server 250 should be relatively certain that this was aphrase, and thus can avoid scoring occurrences of “new york” and “yorkgiants” within the context “new york giants” as potential subphrases.The strong phrase threshold can be adjusted so that all candidatephrases are devolved (e.g., setting the strong phrase threshold toinfinity) or it can be set low (even below the initial phrasethreshold), to limit the subphrase analysis.

Accordingly, each candidate phrase is subdivided into two child phrases,and the phrase identification server 250 applies the following scoringrules to the child phrases, depending on the candidate phrase's score.First, if the candidate phrase's score is below the strong phrasethreshold, the phrase identification server 250 applies the followingrules for scoring the child phrases:

-   -   1. If the candidate phrase hit position was “exact” and had a        score of X, the first child is indicated as an initial hit        position with score X, and the second child is indicated as a        final hit position with score X.    -   2. If the candidate phrase hit position was “initial” and had a        score of X, the first child is indicated as an initial hit with        score X, the second child is indicated as a medial hit position        with score X/2.    -   3. If the candidate phrase hit position was “final” and had a        score of X, the first child is indicated as a medial hit with        score X/2, the second child is indicated as final hit with score        X.    -   4. If the candidate phrase hit position was “medial” and had a        score of X, both children are indicated as medial hits with a        score X/2.

If the candidate phrase's score is equal to or above the strong phrasethreshold, the phrase identification server 250 applies the followingrules instead:

-   -   5. If the candidate phrase hit position was “exact” and had a        score of X, then skip further scoring.    -   6. If the candidate phrase hit position was “initial” and had a        score of X, the second child is indicated as a medial hit with        score (−X/2)    -   7. If the candidate phrase hit position was “final” and had a        score of X, the first child is indicated as a medial hit with        score (−X/2)    -   8. If the candidate phrase hit position was “medial” and had a        score of X, both children are indicated as medial hits with        score (−X/2)

To further illustrate how the set of scoring rules is applied, theprevious example is continued. Assume the initial scoring functionapplied to the word sequences scores 1 point for a body hit, and 2points for a title hit; let both the initial phrase identificationthreshold and strong phrase thresholds be 3 points, so that allcandidate phrases are tested for their subphrases. After processing thedocument text, the phrase map table can be considered as:

Sequence Data “The quick brown fox jumps” 1 occurrence, body text,initial, 5 words “quick brown fox jumps over” 1 occurrence, body text,medial, 5 words “brown fox jumps over the” 1 occurrence, body text,medial, 5 words “fox jumps over the lazy” 1 occurrence, body text,medial, 5 words “jumps over the lazy god” 1 occurrence, body text,final, 5 words “The quick brown fox” 1 occurrence, title text, exact, 4words

For the rest of this calculation, the phrase identification server 250need only points and positions, so this can be rewritten as:

Sequence Initial Score “The quick brown fox jumps” 1 initial point“quick brown fox jumps over” 1 medial point “brown fox jumps over the” 1medial point “fox jumps over the lazy” 1 medial point “jumps over thelazy god” 1 final point “The quick brown fox” 2 exact points

“The quick brown fox jumps” does not exceed the initial or strong phrasethresholds, so the phrase identification server 250 decomposes it intoits two children, “The quick brown fox” and “quick brown fox jumps”, andscores them according to the rules (1)-(4), above. Note that its firstchild, “The quick brown fox”, gets a score of 1, per rule (2) above, andis already in the table, so this additional score is added, as shownbelow. The second child gets added to the table with its score of 0.5,per rule (2), as well.

Sequence Revised Score “quick brown fox jumps over” 1 medial point“brown fox jumps over the” 1 medial point “fox jumps over the lazy” 1medial point “jumps over the lazy god” 1 final point “The quick brownfox” 2 exact points, 1 initial point “quick brown fox jumps” 0.5 medialpoints

When the phrase identification server 250 processes “quick brown foxjumps over,” the same analysis results with its children “quick brownfox jumps” (which receives another 0.5) and “brown fox jumps over”(which is a new entry with 0.5):

Sequence Revised Score “brown fox jumps over the” 1 medial point “foxjumps over the lazy” 1 medial point “jumps over the lazy god” 1 finalpoint “The quick brown fox” 2 exact points, 1 initial point “quick brownfox jumps” 1 medial point “brown fox jumps over” 0.5 medial points

After the phrase identification server 250 is done with all of the5-word phrases, the revised scores are:

Sequence Revised Score “The quick brown fox” 2 exact points, 1 initialpoint “quick brown fox jumps” 1 medial point “brown fox jumps over” 1medial point “fox jumps over the” 1 medial point “jumps over the lazy” 1medial point “over the lazy god” 1 final point

The candidate phrase “The quick brown fox,” has a total score of 3 andso the phrase identification server 250 outputs it, and devolves itaccording to rules (5) and (6): the exact points do not devolve, and theinitial hit score results in a medial hit worth −0.5 points on “quickbrown fox”. Thus:

Sequence Revised Score “quick brown fox jumps” 1 medial point “brown foxjumps over” 1 medial point “fox jumps over the” 1 medial point “jumpsover the lazy” 1 medial point “over the lazy god” 1 final point “quickbrown fox” −0.5 medial points

When the phrase identification server 250 processes “quick brown foxjumps”, its first child's contribution is thus cancelled by the negativecontribution: the only occurrence of “quick brown fox” occurred within astrong phrase. Thus:

Sequence Revised Score “brown fox jumps over” 1 medial point “fox jumpsover the” 1 medial point “jumps over the lazy” 1 medial point “over thelazy god” 1 final point “quick brown fox” 0 points “brown fox jumps” 0.5medial points

The phrase identification server 250 continues through the rest of thephrases, devolving 4-word sequences into 3-word sequences, then into2-word sequences. None of these subphrases will hit either threshold inthis example. The output of stage 1 on this document is therefore asingle candidate phrase: “The quick brown fox”, 3 points. Of course, inpractice a document will yield many candidate phrases.

The phrase identification server 250 repeats these steps for everydocument in the document set that is being used for phrase extraction.The result is an initial set of candidate phrases, and for eachcandidate phrase a list of document phrase scores, from the documents inwhich it was considered a reasonable candidate. The individual documentphrase scores reflect how strongly each candidate phrase was used withineach document. A candidate phrase that appears repeatedly, or insignificant locations with the a document will have a high documentphrase score for that document.

(2) Cross-Document Phrase Merging

The first stage 302 of the phrase extraction process providedinformation about how strongly various candidate phrases are indicatedto be real phrases by individual documents. In the next stage 304, theoverall support for each candidate phrase across the document collectionis determined. For each candidate phrase, the phrase identificationserver 250 combines its document phrase scores into combined score toget a total measure of the support of the candidate phrase within thedocument collection. The combined score can be a sum, average, or otheraggregate value of the individual document phrase scores.

Each candidate phrase is then evaluated to determine whether it is areal, valid phrase to be retained for indexing. This evaluation is basedon the combined phrase score, as well as the set of document phrasescores for the candidate phrase. In general, the phrase identificationserver 250 applies a rule that a candidate phrase is retained where:

-   -   1. It is strongly supported by at least one document (e.g., the        maximum of its individual document phrase scores is above a        first threshold), or    -   2. It is moderately supported by a range of documents (e.g., the        combined score is above a second threshold), or    -   3. It occurs very broadly (e.g., the number of documents from        which it received a minimum score is above a third threshold;).

Strong support is exemplarily shown by a phrase occurring in at leastone document with a high document phrase score, as computed during theinitial extraction stage, that exceeds a first threshold. Moderatesupport is exemplarily shown by the phrase having its combined documentphrase scores being above a second threshold. Finally, broad support isexemplarily shown by the phrase occurring in a minimum number ofdocuments having a minimum phrase score; the minimum required score canbe set at a level to further control or limit the required breadth ofsupport. These thresholds are selected based on the designer's desiredlevel of liberal or conservative inclusion of candidate phrases, andwill also depend on the characteristics of the document set in terms ofthe structure of the documents (e.g., length, formatting, etc.).

The output of this stage 304 is a further revised list of candidatephrases, which are initially deemed to be real and valid phrases,together with the combined score.

(3) Remove Redundant Phrases

A side-effect of the initial extraction stage 302 is that both acandidate phrase and its subphrases may be emitted from the samedocument, if the phrase scored between the initial threshold and thestrong phrase threshold. This can be generally useful, because if thesubphrase occurred independently of the longer phrase in some otherdocuments, this document just gave further support (in stage 304) to thevalue of the subphrase. Accordingly, in this stage 306, the phraseidentification server 250 processes the current candidate phrase list toidentify a phrases that are subphrases another phrase and that have thesame or almost the same composite score (e.g., the subphrase scoreshould be at least 95% of the phrase score). In that case, thesesubphrases are determined to be redundant, and are eliminated from thecandidate phrase list.

(4) Refine Phrase List with Heuristics and Language Models

The list of candidate phrases produced by stage 306 can be furtherrefined with an optional processing stage using heuristics and/orlanguage specific models. In one embodiment, a set of heuristics iscreated that represent knowledge of a fluent speaker. The heuristicrules eliminate candidate phrases that begin with certain words (e.g.,“or”, “and”, etc) and other artifacts from the above screening process.Additionally, other refinement techniques may be applied, such as theuse of generative language models to further analyze the documents andidentify good or bad phrases, for example language specific models ofgrammar and syntax.

Once processed with the final refinement rules and models, the candidatephrase list is ready for use in indexing. This phrase list is updated tothe phrase data 255. Each phrase is assigned a unique phrase numericalidentifier; the phrase identifier can be a hash of the phrase, usingSHA-1 or a similar hash function, to generate a unique value. The phrasedata 255 there is then maintained in a phrase table that contains thephrase identifier, along with the phrase itself, sorted by the phraseidentifier. This arrangement allows for very fast checking of a wordsequence to determine if it is a known phrase. In addition, for eachphrase, an expected probability of the phrase in the index is maintainedand updated, based on the frequency of the phrase and the number ofdocuments in the index.

As noted above, the document set used for phrase extraction can be anyset of documents in the document collection. A further set documentsthat can be used for this process are the search query logs for thesystem 100, which comprise a large set of search queries. In this case,text breaks are indicated by syntactic objects in the query (e.g.,quotation marks, Boolean operators such as AND, NOT, and OR, and similaroperators) or other visual separations such as punctuation marks. Theresults of the extraction process on this document set can be combinedwith those of other document sets during the combination operation ofstage 304; in that case, hits from query logs are assigned a lowerweight in the combination algorithm than hits from documents.

In a typical embodiment of the phrase extraction process, anywhere from1,000,000 to 10,000,000,000 phrases can be identified.

2. Indexing System with Tiers and Shards

The next functional operation of the indexing system 110 is to indexdocuments in the collection using the phrase data 255, and store theindex information in the segment shard files 225, from which the indexservers 200 will be updated. This process of the indexing system 110 ismanaged by the indexing master server 260, which structures the indexwithin the index server cluster 160 in such a way as to take advantageof the enhanced semantic information that is provided by the use ofphrases to index the documents to both reduce the overall storagerequirements for the index, as well as to structure this information toimprove query processing times.

Generally, the indexing process has two basic stages:

1) Generating phrase posting lists for a set of documents; and

2) Assignment of phrases and documents to index servers.

In the first stage, the phrase posting lists are generated by theindexing server master 260 by processing a set of documents from thedocument collection, and in each document, identifying which phrases arerelated to the document. For each identified phrase, the posting listfor the phrase is updated to indicate the document, and informationabout the significance of the phrase to the document. This significanceinformation can include an information retrieval score for the phrasewith respect to the document, as well as other information; theinformation retrieval scoring algorithm can be the same as that usedduring ranking of search results, or a different algorithm. In eithercase, the significance information is available to the search system 110for use in ranking search results. This process is repeated across thedocument set for the indexing pass. Typically, an indexing pass cananalyze between one and billions of documents, as desired by the systemimplementer.

This process can be implemented as follows, for a given document. Theindexing server master 260 traverses the words of the document with asliding phrase window of length n, where n is a maximum phrase length.The maximum phrase length is typically set to the length of the longestphrase in the phrase data 255. The length of the window will typicallybe at least 2, and preferably 4 or 5 terms (words). The indexing servermaster 260 checks each sequence of words (from length n to length 1)within the phrase window against the phrase data to determine if it ispresent as a known phrase. This is done by generating the hash of theword sequence, and checking it against the phrase table in the phrasedata 255. The sequences that are considered as potential phrases includeall subsequences of consecutive words within the window, multi-wordsequences including one or more skipped words, and sequences includingpermutations (reorderings) of two or more words. Most generally, giventhe phrase window of length n, all n!(Sum(1/k!)) permutations, where thesum over k goes from 1 to n, are evaluated as potential phrases. Aphrase window may be terminated by an end of line, a paragraph return,the end of the document, a markup tag, or other indicia of a change incontent or format.

For each phrase that is identified within the document, the indexingserver master 260 determines information about the significance of thephrase within the document. The indexing server master 260 can usevarious measures of significance, including maintaining a count of thefrequency of occurrences of the phrase within the document, the positionof the occurrences. The updated phrase posting lists are stored to thesegment shard files 225, as will be further explained below, from whichthey will be copied to the index servers 200.

In one embodiment, this significance information for a given documentand a phrase is an information retrieval score, generated by aninformation retrieval scoring algorithm, in which the phrase is used a“query” and the document by itself is used as the search corpus. Thisapproach generates an information retrieval score of the relevance ofthe phrase to the document itself. The information retrieval scoringalgorithm can be the same as or different from the one used by thesearch system 110 for final document scoring. Preferably, theinformation retrieval score generated during indexing is combined withinformation retrieval score generated during query processing that isbased on a received query and some subset of the documents based on thequery; the final score for each document is then a function of these twotypes of scores. One implementation of the scoring model is a bifurcateddocument relevance scoring model that uses two scoring functions, afirst scoring algorithm (used at indexing time) that generates thephrase relevance score for a document, and a second scoring algorithm(used during query processing) that takes as input a set of phraserelevance scores (for phrases of the query) along with the query itselfand optionally, and generates a final relevance score for a givendocument. In one embodiment of this approach, the posting list for eachphrase is limited to include only the document identifier of the currentdocument and the information retrieval score for the phrase with respectto the document; other significance information is not stored in thephrase posting list itself. This is beneficial to reduce the overallsize and thereby memory requirements for the phrase posting list.

Once all of the documents in the indexing set have been processed, thenext stage is the assignment of phrase posting lists to the indexservers 200 for storage and serving. This is done by generally groupingthe phrase posting lists into tiers and partitioning the phrase postinglists into shards, which are then stored in various index servers 200.This storage approach will be described by way of a systematicexplanation of various embodiments, as follows.

For the simplest embodiment of this storage process, the index servercluster 160 is logically structured as shown in FIG. 4. In FIG. 4 thereis shown the index server cluster 160 with various index servers 200.For purposes of explanation, there are P phrases in the phrase data 255,and hence P phrase posting lists. Each phrase posting list includes anumber of document identifiers therein.

In this first embodiment, each of the P phrase posting lists ispartitioned into a number (S) of portions called “shards”; thepartitioning process can be called “sharding.” Each posting list shardwill ultimately be stored by one (or more) of the index servers 160. Forpurposes of explanation, during indexing a posting list shard will becalled a segment shard because it will be stored to the segment shardfiles 225; once an indexing pass is completed, the segment shard filesare copied over to the index shard files 165 and form correspondingindex shards (and thus the current index), and then copied to the indexservers 200 for serving. An index shard may be stored by an index server200 in memory, on disc, in a combination thereof. Thus, in FIG. 4, thereis shown index server 200.1 through 200.S; for example, if S=1000, thenthere will be 1000 shards of each phrase posting list. Index server200.1 stores the first shard, shard 1, for each of the phrase postinglist, lists 1 through P; index server 200.2 stores shard 2 for all ofthe phrase posting lists, and so forth through index server 200.S, whichstores the S^(th) shard for all phrase posting lists. For clarity, itshould be noted that an index server 200 operates as server program in acomputer, and a given computer can support multiple index servers 200;thus a given server computer can store the phrase posting lists formultiple shards (i.e., shards S_(i) for some set of values of i<=S).Additionally, a given shard (e.g. one containing high frequency phrases)may be duplicated and stored on multiple index servers 200 in order toincrease performance.

The partitioning of the phrase posting lists into the shards is based ona document assignment function, which assigns documents within a phraseposting list to individual one of the shards, such that a given documentis always assigned to the same shard (and hence index server 200)regardless of which phrase posting list the document appears in. Forexample, if a given document appears in the phrase posting lists of 50different documents, it will still reside in the same shard by virtue ofthe shard assignment function. Within each shard, each of the phraseposting lists is then ordered by document identifier.

In one embodiment, the shard assignment function is a modulo function ofthe document identifier and S, the number of shards to be used:

Shard_ID=Document_Identifer(Mod S).

The shard assignment function is flexible in that the particularassignment operation can itself be a function of the phrase that isbeing sharded. In other words, instead of a single shard assignmentfunction, multiple different shard assignment functions can be used, andthe selection is based on the particular phrase whose posting list isbeing shared. More particularly, phrases that have very large postinglists (e.g., on the order of millions of documents) can be sharded intoa large number of shards, whereas phrases with smaller posting lists(e.g., hundreds of documents), can be sharded with a smaller value of S.The selection of shard assignment function is then determined bydocument collection statistics, such as frequency or probability of thephrase.

The following simplified example illustrates the basic concept ofsharding, using a single shard assignment function.

Assume the following phrase posting lists P1, P2, P3, and P4, each ofwhich has some set of documents, indicated by document

Phrase No. Document IDs P1 3 5 7 11 17 26 30 35 36 37 39 41 43 45 49 P20 1 3 8 11 17 26 28 31 38 39 43 45 46 49 P3 4 5 7 8 9 13 17 22 30 32 3537 38 46 48 P4 3 4 6 9 11 15 22 26 30 31 33 37 38 41 45

Further assume that there are 4 shards, S0 through S3. Then the moduloshard assignment function results in the following distribution of thephrase posting list documents to shards:

P#, S# Doc. Id P1, S0 36 P2, S0 0, 8, 28 P3, S0 4, 8, 32, 48 P4, S0 4P1, S1 5, 17, 37, 41, 45, 49 P2, S1 1, 17, 45, 49 P3, S1 5, 9, 13, 17,37 P4, S1 9, 33, 37, 41, 45 P1, S2 26, 30 P2, S2 26, 38, 46 P3, S2 22,30, 38, 46 P4, S2 6, 22, 26, 30, 38 P1, S4 3, 7, 11, 35, 39, 43 P2, S43, 11, 31, 39 43 P3, S4 7, 35 P4, S4 3, 11, 15, 31

In this table example, it can be readily seen that a given document isalways assigned to the same shard. Each of these shards is then storedin one or more of the index servers 200. In this simple example, fourindex servers 200 would be used, one for each shard.

One benefit of sharding phrase posting lists is that it allows multipledifferent index servers 200 to operate in parallel during queryprocessing, without requiring them to cross-communicate their queryprocessing results. This benefit arises because the index servers thathold the different shards of a given phrase posting list can all operateconcurrently to serve their document IDs to other index servers forquery intersection, without having to wait on each other.

FIG. 5 illustrates a next embodiment, introducing the concept of tiersof index servers 200. A number of tiers is selected for a givenimplementation. In typical embodiments, there are between 2 and 10tiers. Each phrase posting list is then assigned to one tier of indexservers 200. The assignment of phrase posting lists to the tiers isbased on a phrase assignment function, which will be further describedbelow. It will be appreciated that the tiers are logical assignments ofthe index servers 200 that governs their functional roles, and notphysical tiers in which the computers hosting the servers are located.Nonetheless, the tiers can be illustrated, as in the figures, as beingseparate from each other, which of course is not necessary in practice.

For example, in FIG. 5 there is shown two tiers, Tier 1 and Tier 2, eachof which is associated with some set of index servers 200. The entiretyof the phrase posting lists are divided amongst the two tiers, so thatsome phrase posting lists are assigned to the index servers 200 in Tier1, and the remaining phrase posting lists are assigned to the indexservers 200 in Tier 2.

Within each tier the phrase posting lists therein are divided into theshards as described above with respect to FIG. 4. The number of shardsin each tier is variable; as shown in FIG. 5, there are S1 shards inTier 1 and S2 shards in Tier 2. In one embodiment, the number of shardsin tier n is an integer multiple of the number of shards in tier n−1:

S _(n) =kS _(n−1)

where S_(n) is the number of shard in tier n, and k is an integer. Thisrelationship between the number of shards in the tiers is beneficialduring query processing time because it constrains a given index server200 in tier n to communicate with at most k index servers in tier n+1during query processing, rather than all of the index servers in tiern+1. Each index server 200 knows the number of shards (and hence indexservers 200) in the next tier, and thus can readily determine via theshard assignment function which index servers 200 in that next tier areto receive its shard during query processing.

FIG. 6 shows an example of a three tier embodiment, here using threetiers, Tier 1, Tier 2, and Tier 3. Tier 1 stores a set of 10,000 phraseposting lists, each of which has one shard. These 10,000 phrase postinglists are distributed across a selected number of index servers 200,which can be for example, 10 index servers 200, each storingapproximately 1,000 phrase posting lists (the illustration of 1,000phrase posting lists per index server 200 in FIG. 6 is merelyillustrative, in practice the actual number of phrase posting lists perindex server 200 will vary). Tier 2 stores 1,000 phrase posting lists,each of which is partitioned into 10 shards. Here, in this tier eachindex server 200 stores the entirety of a shard for its assigned phraseposting lists. Note as well that the number of shards here is an integermultiple of the number of shards in tier 1. Finally, tier 3 stores 100phrase posting lists, but across 1,000 shards. Again, the number ofshards in this tier is an integer multiple of the number of shards inthe previous tier.

The assignment of phrases to tiers can be implemented in various ways.In one embodiment, the assignment of phrases to tiers is as follows.First, a query processing cost measure is selected which represents somecost for the index servers 200 when processing queries against a phraseposting list. The cost measure may be a direct measure of processingtime, communication times or bandwidth, memory requirements, or thelike. The cost measure may also be an indirect measure, based onattributes of the phrase posting lists, such the length the phraseposting list in terms of number of documents, number of bytes, or otherfactors. A phrase assignment function assigns a phrase to a tier usingattributes of the phrase posting list, such as its length, and thecapacities of the available index servers (e.g., their available memory,processing speed, and the like). For example, phrase posting lists thatwill have high processing costs can be assigned to tiers having thehigher performance index servers (e.g., faster, more memory, etc.). Thisarrangement is beneficial since during query processing, the phrases inthe query can be selectively processed only by those index servers 200in the tiers that contain the phrase posting lists for those phrases. Aphrase assignment map is maintained identifying for each phrase to whichtier, shards, and index servers 200 within the tier the phrase isassigned, along with additional information about the phrase, such asits frequency in the document collection, ranking information forcomputing a ranking score, or the like.

In one embodiment, the phrase posting lists are assigned to the tiers byassociating each tier with a minimum cost (in terms of the above costmeasure), and then assigning a phrase posting list to the tier for whichthe cost of the phrase posting list is greater than the minimum cost forthat tier, but less than the minimum cost for the next tier. Once aphrase posting list is assigned to a tier, it is stored in one or morethe index servers 200 in the tier, as illustrated in FIGS. 4-6.

One particular embodiment of the cost measure is the length of thephrase posting list. This is used as an indirect measure of queryprocessing costs, as larger phrase posting lists require a greaternumber of index servers 200, and hence an increased amount of potentialinter-server communication during query processing. Each tier then isassociated with a minimum phrase posting list length L, such that L iszero for the first tier, and L for each M^(th) tier (M>1) is greaterthan L for the (M-1)^(st) tier. The length of a posting length can bethe number of documents therein, number of bytes, or other measures.Each phrase posting list is assigned to a tier based on its length, byassigning each phrase posting list to the tier in which its length will“fit” between the minimum length of that tier and the minimum length ofthe next tier. The index servers 200 in each tier store the phraseposting lists assigned to that tier.

In one embodiment, index servers 200 in the first tier can be used tostore phrase posting lists for relatively low frequency phrases; thesewill generally (though not necessarily) be relatively longer phrases,such as “Harry Potter and the Order of the Phoenix”, or rather obscureword combinations, such as “psychedelic popsicles”. Typically, are alarge number of such phrases, but each has a rather short phrase postinglist (e.g., on the order of <1,000 documents). Accordingly, appropriatethresholds can be assigned to tiers 1 and 2 (e.g., 0 for tier 1, and1,000 for tier 2), and all phrase posting lists with less than 1,000documents therein are assigned to tier 1. Since each phrase posting listis short, it can be stored in a single shard that is not furtherpartitioned. Index servers in next tier(s), can store progressivelylonger phrase posting lists, using the appropriate limits on the tiersand the phrase assignment function. The last tier, i.e., the M^(th)tier, can be used to store the phrase posting lists for very highfrequency phrases. In a typical document collection there will arelatively small number of these, but each will have a phrase postinglist on the order of millions of documents. Here then a large number ofshards is used to partition the phrase posting lists.

From the foregoing, the use of tiers and shards can be now explainedmore generally. The phrase posting lists may be considered to be anarray of rows and columns. Each row corresponds to a single phraseposting list, and column corresponds to an i^(th) document in the list.Tiers then are understood to group the rows together (though notnecessarily in any particular order within each group); this can beconsidered a “horizontal” grouping. Shards by contrast can be understoodas “vertical” partitions, since they divide up each row into a number ofportions. Several insights can be gained from this analysis. First, thegrouping by tiers and the partitioning by shards can be independentlycontrolled, or one can be made dependent on the other, as shown above.Second, this independent control can be exercised to a very granulardegree, so that each phrase posting list can be very selectivelyassigned to a tier and a number of shards. Third, the types ofassignment functions for the tiers and shards can be selected tooptimize different performance characteristics of the index servers 200.Thus, the phrase assignment function can be used to optimize for oneaspect of performance, say communications costs, and the shardassignment function can be used to optimize for a different aspect ofperformance, such as increased parallelism amongst the index servers.

3. Index Maintenance

The foregoing section described the processes and structures by whichthe phrase-based index is created. Further aspects of the presentinvention are the processes and structures used to maintain and updatethe index over time. Referring again to FIG. 2, the indexing servermaster 260, server cluster master 220, swap master server 240, indexshard files 115, and segment shard files 225 are the componentsprincipally involved in updating the index.

As described above, the index for the document collection is dividedinto a number of tiers and shards, which are then served by the variousindex servers 200. To facilitate updating the index an additionalorganizational structure is used, called a segment. The set of alldocuments in the index is divided into a number of segments; thesegments are independent of the tiers and shards described above. Eachdocument is typically represented in only one of the segments, though insome instances a document may appear in multiple segments. For example,if the document is an important document that should be indexedfrequently, then this document can be placed into multiple segments. Inthis case, the document may be assigned a different document identifierfor each instance, and when the document is retrieved in search results,only the most recent instance of the document will be returned. A givensegment will contain documents that are indexed in various phraseposting lists across the tiers and shards. Typically a segment willstore the information for 1M to 10B documents. A typical implementationwill have between 10 and 1000 segments, depending on the number ofdocuments in the document collection.

As described above, the indexing server master 260 creates the index byprocessing a set of documents from the document collection, and in eachdocument, identifying which phrases are related to the document. Moreparticularly then, indexing server master 260 performs this processingon each of the segments, creating a segment phrase index of phraseposting lists for the documents in the segment. This segment phraseindex is then tiered and sharded as described above to formsegment-based shards, or more simply the segment shard files 225, asmentioned referenced above. Each segment shard file is stored under themanagement of the server cluster master 220. The server cluster master220 maintains an index specification, which identifies for each segmentshard file, version information (e.g., version number, date of lastupdate), and location information identifying the machine anddirector(ies) storing the segment shard file, and the correspondingindex shard file to which the segment shard file is associated.Additional update data optionally may be included that more granularlyspecifies the differences between the current segment version and theprevious version in terms of which phrase posting lists in the shardwere updated.

To complete the update of the index servers 200, the segment shard files225 from various segments are combined into the index shard files 115which are served from the index servers 200. To do this, the indexspecifications for all segments are read by the server cluster master220, and locations of the segment shard files corresponding to the mostrecent versions of each segment are determined. Next, the segment shardfiles for each index shard file are combined.

FIG. 7 conceptually illustrates the various embodiments by which thesegment shard files are merged into index shard files. In oneembodiment, this is done by a separate set of merging servers 702, whichread, for each index shard file 704, the corresponding segment shardfiles into memory, combine each phrase posting list therein to form amerged phrase posting list, and write the resulting merged list as anindex shard file 704 back to the server cluster master 220, which storesthem in the update index 230. The individual index servers 200 then copythe new index shard files from the update index 230 into their ownmemory/disk. In an alternative embodiment also illustrated in FIG. 7,the index servers 200 themselves perform the combining operation,reading the segment shard files from the update index 230 into their ownmemory, and doing the merge operation locally. The advantage of thelatter embodiment is that the additional work of copying the mergedindex shard files (which are larger than the segment shard files) fromthe update index 230 to the index servers 200 is not needed, but thedisadvantage is that additional work must be performed by the indexservers 200. The selection of which approach to use can be made as partof system design, or even at run time, so that index servers 200 thatare lightly loaded are tasked with their local merges, whereas forheavily loaded index servers 200, the merging servers are used.

The swap master server 240 directs the merging process in which themerges are performed by the individual index servers 200. The swapmaster server 240 is notified each time a segment update is stored tothe server cluster master 220. For each such update, the swap masterserver 240 determines from the index specification which index shardfiles 115 need to be updated on the index servers 200, based on theversion information for the segment shards 225. The index specificationis used then to determine which index servers 200 are associated withthese index shards. This set of index servers 200 then forms the currentupdate group.

The swap master server 240 instructs the index servers 200 in the updategroup to begin the merge process. This process is typically done duringa window of low load activity, such as late at night. The swap masterserver 240 provides each index server 200 in the update group withinformation identifying where the segment shard files 225 for its updateare held (which machine and directory). Each index server 200 isresponsible to retrieve the updated segment shard files 225, merge thesewith the phrase posting lists in the existing current index shards, andnotify the swap master server 240 upon completion. As noted above, the aseparate set of merge servers can perform the merging operation directlyon the server cluster master 220, in which case they are controlled bythe swap master server 240 in the manner just described. Additionally,the merge servers can be used in this fashion when creating an indexfrom scratch, otherwise using the index servers 200 during regularupdates.

After all of the phrase posting lists have been merged relative to theupdated segment, the swap master server 240 then notifies each of theindex servers 200 to swap its currently served index shards (in memoryor disk) with the updated and merged index shard files 115. Each indexserver 200 manages the swap process as follows. For shards that areserved by multiple index servers 200, the swap master server 240instructs the index servers 200 corresponding to multiple copies of eachindex shard file to swap at different times, so that at any given time aminimum number of copies of each index shard file are available forquery serving from index servers 200 that are not swapping. This is doneso that the index servers 200 can continue to service queries fordocuments in the phrase posting lists of the older shards. For indexshards which are served by only one index server 200, the swap masterserver 240 instructs the index servers 200 to swap index shard filescorresponding to different shards of a single posting list gradually, sothat at any given time no more than a small percentage of shards, e.g.5%, of each such phrase posting list are in the process of swapping.Preferably, phrase posting lists served by single index server 200 aredivided into enough shards that a single shard contains no more thanthis desired percentage of phrases or documents therein. As a result,during the write passes only a very small portion of the overalldocument index will not be served, thus having little impact on theoverall search results.

III. Search Query Processing

As described above with respect to FIG. 1, the front end server 140receives a query from the user interface server 130, and in cooperationwith the search system 110 creates a set of search results to beprovided to the user interface server 130. FIG. 8 illustrates theoverall data and process flow for the processing of search queries ashandled by the front end server 140. Generally, the process takes asinput a Boolean word tree 801 from the user interface server 130,wherein the leaf nodes include individual words from the query and thenon-leaf nodes contain Boolean operators AND, OR, or NOT. The Booleanword tree may also include, for any of the query words (or groupsthereof), additional nodes that are synonyms and related words, asderived from any external synonym source (e.g., any source from simplethesaurus tables to concept-based analysis algorithms), in which case anode may include an annotation that indicates whether it is part of theoriginal query or derived from a synonym source. The queryphrasification module 810 is responsible for decomposing a query treeinto a phrase tree 815 comprising a set of phrases and the semantics ofthe query. The query scheduling module 820 then optimizes phrase treeinto a query schedule 825 by which this phrase tree is executed. Thequery execution module 830 then manages the execution of the queryschedule by the index servers 200. The index servers 200 create a set ofresults (documents) which are returned to the front end server 140,which in turns creates a final search result set 835 and provides it tothe user interface server 130. The following sections describe thesestages of operation in detail.

1. Query Phrasification

FIG. 9 illustrates the process flow of the query phrasification module810 according to one embodiment. Other embodiments can have differentand/or additional stages than the ones shown in the figure. The queryphrasification module 810 takes as input the Boolean word tree 800; thisBoolean word tree can be of any complexity, including any number ofconjuncts and disjuncts, including nested groups. The queryphrasification module 810 generates as output a Boolean phrase tree,wherein the leaf nodes contain one or more phrasification (eachcontaining one or more phrases) resulting from the phrasification andthe non-leaf nodes contain the Boolean operators AND, OR and NOT. Itshould be noted that the query phrasification process is not limited touse with search queries as inputs, but can be applied in many otherdifferent applications to any input comprising a set of words and(optionally) operators; for example the query phrasification process canbe applied in any type of natural language processing, text mining,document analysis, including routing, description generation,categorization, concept matching and extraction, and so forth. Thestages of phrasification are as follows.

The query phrasification module 810 receives from the user interfaceserver 130 the Boolean word tree 800. The query phrasification module810 restructures 902 the word tree using de Morgan's laws into anequivalent tree comprising a single top level OR node, each of thedisjuncts being an AND of a number of leaf nodes and/or NOTs of leafnodes. This process can be described as flattening the tree. Forexample, the word tree:

(A OR B) AND (C OR D)

is restructured to:

(A AND C) OR (A AND D) OR (B AND C) OR (B AND D)

where A, B, C, and D are nodes of the word tree. As can be appreciated,each of these nodes could as well be complex nodes as well, with furtherchild nodes, which are in likewise restructured.

The query phrasification module 810 then generates all possiblephrasification from the restructured word tree. This is done by takingeach subtree—which is itself a disjunct of the top-level OR—of the form(A AND B AND C . . . M) and creating all possible partitions of theconjuncts A, B, C . . . M into disjoint phrases. For example, thesubtree:

(A AND B AND C AND D)

generates the following phrases phrasifications, where each group ofterms in quotes is a phrase:

“AB CD”

“A” AND “B C D”

“A. B” AND “C D”

“A B C” AND “D”

“A” AND “B” AND “CD”

“A” AND “B C” AND “D”

“A B” AND “C” AND “D”

“A” AND “B” AND “C” AND “D”

A more concrete example is “New York Ethiopian restaurants”, whichyields the following phrasification:

“New York Ethiopian restaurants”

“New” AND “York Ethiopian restaurants”

“New York” AND “Ethiopian restaurants”

“New York Ethiopian” AND “restaurants”

“New” AND “York” AND “Ethiopian restaurants”

“New” AND “York Ethiopian” AND “restaurants”

“New York” AND “Ethiopian” AND “restaurants”

“New” AND “York” AND “Ethiopian” AND “restaurants”

The possible phrasification can be extended to include all permutationsof the order of the conjuncts as well (e.g., phrasification where thesequence of A, B, C . . . is altered).

If there are NOTs in the query, these are treated treat those as beinghard boundaries for the purpose of phrasification. For example, if thequery is:

(A AND B AND NOT C AND NOT D AND E AND F)

then no phrasification are generated that cross the B/C, C/D, or D/Eboundaries, so that only “A B” vs. “A” “B” and “E F” vs “E” “F” areconsidered as potential nodes. This method may be extended by addingother such boundary points within the query, based on annotations in theoriginal Boolean query of words. For example, if the user typed explicitquotes in the query:

“New York” fast food

then no phrases would be derived from “York fast”.

Next, the query phrasification module 810 scores 906 each phrasephrasification, using the expected probability of each phrase in thephrasification, as obtained from the phrase data 255. In some cases, theprobability for a phrase is zero since it is not contained in the phrasedata, and hence is not a real phrase. For example, the phrase “YorkEthiopian” is itself not a phrase, and its frequency would be zero.

The query phrasification module 810 then scores each phrasephrasification using a phrase scoring function. The phrase scoringfunction is designed to trade off precision versus recall in terms ofselecting a final set of phrases. The general model for a phrase scoringfunction is as follow:

$S = {{f(N)}{\prod\limits_{i = 1}^{N}\; {{P\left( p_{i} \right)} \cdot {C\left( p_{i} \right)}}}}$

where,

-   -   S=score for a particular phrase phrasification;    -   N=number of phrases in the phrase phrasification;    -   p_(i)=phrase_(i) in the phrase phrasification where i ranges        from 1 to N;    -   P(p_(i))=probability of phrase_(i), which can be an estimated        probability or derived from the number of documents in the        corpus that contain phrase_(i) divided by total number of        documents in the corpus;    -   C(p_(i))=confidence in the phrase_(i) where C is 1 for phrases        that were in the original input and C<1 for phrases that are        derived from other sources, such as synonym sources that can        provide a scaled measure of the confidence of the phrase; and    -   f(N)=function of N that adjusts the bias between precision and        recall.

In one embodiment, the function f(N) that adjusts the bias betweenprecision and recall is defined as,

f(N)=(β^(N) /N ^(1+α)),

where

-   -   α=an adjustable constant such that α>0 to adjust precision of        phrase phrasification (higher a increases the precision and        reduces the recall), and    -   β=an adjustable constant such that 0<β<1 to adjust the bias        against    -   obtaining too many phrases (smaller β reduces the recall)

The α and β parameters are adjusted by the system designer to trade offthe precision and recall of the resulting search for the phrasephrasification.

Once the phrasification are scored, the query phrasification module 810selects some subset of the highest scoring phrasification. The subset ofselected phrasification can vary from one (i.e., highest scoring phrasephrasification) to some fixed number (e.g., top 10 scoringphrasification), or a selected percentage (50%) of the top scoringphrasification. In addition, the size of the subset can be varied basedon other system variables, such as system load, or input context (e.g.source of the query).

The selected phrasification are organized as a Boolean phrase tree 815having a root OR node, with each of the selected phrasification being adisjunct of the root node.

2. Query Scheduling & Query Optimization

FIG. 10 illustrates the stages for query scheduling 1010 and queryoptimization 1020 handled by the query scheduling module 820, afterquery phrasification and prior to query execution. The input to thequery scheduling module 820 is the Boolean phrase tree 815 previouslydescribed; the output is the query schedule 825 that sequences thephrases in the phrase tree for execution and identifies which indexservers 200 are to be used for the execution operation. The queryschedule 825 can be stored as a Boolean schedule tree which issemantically equivalent to the original tree, and each of whose nodes isadditionally annotated to identify the set of servers on which it shouldbe executed. The query schedule can likewise be stored in equivalentstructures so long as the underlying semantics and schedule informationare maintained.

The query scheduling module 820 executes the following steps forscheduling 1010, as summarized in FIG. 10. For purposes of discussion,the order of child nodes of any Boolean node is assumed to be left toright based on some assigned value (i.e., the left most child node hasthe lowest value).

1) Normalize (1012) the phrase tree by eliminating ANDs of ANDs and ORsof ORs. This is done by collapsing the terms into one another usingassociativity, e.g. ((“a” AND “b”) AND “c”) =(“a” AND “b” AND “c”)) andby eliminating AND and OR nodes with only a single child.

2) Recursively traverse the normalized phrase tree, in depth-firstorder, assigning (1014) to each node a query cost and a plurality ofindex servers 200 according to the following scheduling rules:

-   -   a) To each phrase node, assign the set of index servers 200        associated with the phrase, and a query cost.    -   b) To each NOT node, assign the set of index servers 200 and the        query cost associated with the (unique) child of the node.    -   c) For each AND node:        -   (i) Sort the children nodes in ascending order according to            their query costs, such that the leftmost child node of the            AND is the one with the least cost;        -   (ii) Assign to the AND node the same set of index servers            200 and cost as the child node of the AND with the least            query cost.    -   d) For each OR node:        -   (i) If the OR node is the child of an AND node and is not            the leftmost child thereof, assign to the OR node the same            set of index servers 200 and cost from the node which is            immediately to its left;        -   (ii) Otherwise, assign to the OR node the same set of index            servers 200 of its greatest cost child, but assign it a cost            equal to the sum of the costs of its children.

The above scheduling rules make use of the query costs that werepreviously described. This cost measure can specific to phrases, forexample the length of its posting list, or more generalized such as acost measure associated with the phrase's tier, or even the tier numberitself as a proxy for cost. The cost and index server information isaccessed from the phrase assignment map. The use of the query costsallows the query scheduling module 820 to schedule the which indexservers 200 are used in which order, in order reduce overall processingcosts. The rationale for the remaining rules is as follows.

Scheduling rule (c) is the AND rule. An AND node is given the cost ofits least cost child node because after the first child node isevaluated (e.g., the phrase posting list for the phrase is retrieved),each successive output of the AND is guaranteed to have less than orequal number of documents as this first child. Hence evaluating theleast cost child first (and the remaining children in ascending costorder) means that the evaluation of the AND starts with shortest phraseposting list being sent from the first child's index servers 200 to thenext child's index server 200 for intersection, and that each successiveintersection list will be no longer than the prior one. This reducesoverall communication costs (amount of data being transferred betweenindex servers 200) as well as ensuring the fewest possible number ofintersections of phrase posting lists need to be performed for the ANDnode.

Scheduling rule (d) is the OR rule. An OR node is evaluated by sendingparallel execution requests to the index servers 200 assigned to each ofthe OR node's children. Accordingly, these requests can be sent fromwhichever index server 200 is responsible for the OR node itself. Theleft sibling rule (d)(i) is used where the OR node is a child of an AND,because an OR node itself cannot be the child of an OR and a NOT canonly have one child. In this case, the AND rule has the result thatexecution will be at index server assigned to the AND nodes least costchild; this index server is the server for the OR node's left sibling.Accordingly, it is most efficient (lowest cost) to maintain execution onthis particular index server 200. If this is not the case, it is mostefficient to center evaluation of the OR node so that the most expensivechild is local, and the less-expensive children need to transmit theirdata over the network. The assigned cost is the sum of the OR's childrenbecause the number of phrase posting list entries (documents) that areexpected to be returned is the sum of that number for the children.

3. Query Optimization

The goal of query optimization 1020 is to restructure the query tominimize computational and network costs. This is done by evaluating1022 the structure of the schedule tree using a query cost analysis toselect between logically equivalent versions of various nodes of theschedule tree. The versions that have the lowest costs are selected forthe final schedule tree to be executed. After a restructuring 1022 anoptional rescheduling on the restructured nodes may be performed,depending on which query costs analysis is being used. An implementationof this process by the query scheduling module 820 is as follows.

The query scheduling module 820 first executes a breadth-first traversalof the schedule tree, and evaluating each node therein. Where thecurrent node is OR node with at least one AND child node, such thateither of the following node forms are matched:

-   -   (a) One of the children of the AND is identical to one of the        children of the OR, i.e., having the form (A OR (A AND B)); or    -   (b) There are at least two AND children of the OR such that each        of the AND children itself has a child in common; i.e., having        the form (A AND B) OR (A AND C),

then a basic optimization step is attempted. The query scheduling module820 evaluates cost of the de Morgan inversion of the node to generalform ((A AND (B OR C)) using a cost evaluation function. Whichever formof the node has the lower cost (the original OR node or the resultingAND node), is kept in the final schedule tree. The particular costevaluation function can be selected by the system implementer, based onwhich query cost measure is being used (e.g., tier based cost, phraseposting list cost, etc.). Three different cost evaluation functions arenext described, any one of which may be used to make this decision solong as it is used consistently. The following discussion refers to FIG.11A showing an OR node whose children are AND nodes, each of which havecommon children Y_(i) along with other (unshared) children X_(ij). Thede Morgan inversion would replace this tree with the logicallyequivalent tree shown in FIG. 11B.

Tier-based cost: Where the query costs are a function of the tier for agiven phrase, then the de Morgan inversion is performed if and only if,

minimum[cost(Y_(i))]≦minimum [cost(X_(ij))], or if

minimum[cost(Y_(i))]≧maximum [cost(X_(ij))].

This means the inversion will be performed where the cost for processingthe common children node Yi is either less than the minimum of the costsfor the unshared nodes X_(ij), or greater than the maximum cost of thesenodes. The inversion is desirable in this case because it will result inthe AND of the Y_(i) being evaluated en masse and sent to the X_(ij) (inthe former case) or the OR of the X_(ij) being evaluated and sent to theY_(i) (in the latter case), in both cases eliminating the duplication ofeffort in evaluating the Y_(i). If neither of these conditions weresatisfied, then evaluating the inverted tree would result in evaluatinga higher-cost node (the AND of the Y_(i)) prior to a lower-cost node(the least of the X_(ij)) and transmitting the longer posting list overthe network, which would be inefficient. If a node is restructured basedon outcome of the tier-based cost evaluation, an optional rescheduling(1010) and optimization (1020) of the restructured node can be done;this is useful because the updated tier assignments will be used as therecursive optimization progresses.

Phrase posting list length-based cost: Where the query costs are thelengths of posting list lengths (or a function thereof), then the deMorgan inversion is performed if and only if,

minimum [cost(Y_(i))] ≤ minimum  [cost(X_(ij))], or  if${{minimum}\;\left\lbrack {{cost}\left( Y_{i} \right)} \right\rbrack} \geq {\sum\limits_{i = 1}^{n}\; {{{cost}\left( X_{ij} \right)}.}}$

This means that the inversion will be performed where the cost ofprocessing the factored node Y is either less than the minimum of thecosts for the disjunct nodes X, or greater than the summed cost of thesenodes. The rationale is the same as in case (i), but because the costsbeing used are now the same as those used during scheduling, we can usea better expression for the cost of the OR of the X_(ij). A reschedulingof the restructured node is not necessary in this case.

Subtree cost: A third query cost measure is an evaluation cost of anysubtree of a scheduled query tree, which approximates the network loadthat would be generated by executing that subtree. Such a cost may becalculated as follows:

-   -   a) Perform a depth-first recursive descent of the subtree.    -   b) The evaluation cost of a phrase node is equal to its one of        the cost (e.g. phrase posting list length or other a priori cost        measure).    -   c) The subtree cost of an AND node is defined as follows. As        noted above, the children nodes of an AND node are ordered from        least to most costs. Beginning then from the least cost child        node, the child nodes are partitioned into subsequences, each        subsequence containing a set of children which have the same        assigned set of index servers 200. To each subsequence, assign        an evaluation cost equal to the minimum evaluation cost of any        child node of that subsequence. The evaluation cost of the AND        node is the sum of the costs of its subsequences.    -   d) For an OR node, an initial evaluation cost of the OR node is        assigned that is equal to the sum of the evaluation costs of its        children. If the OR node is the child of an AND node, and it is        not a member of the AND node's leftmost subsequence, then the        evaluation cost of the OR node is the lesser of its initial        evaluation cost and the evaluation cost of the subsequence        immediately to the left of it. Otherwise, the evaluation cost of        an OR node is equal to its initial evaluation cost.    -   e) The evaluation cost of a NOT node is equal to the evaluation        cost of its child.

The subtree cost function has the effect that the de Morgan inversion isbe performed if and only if the evaluation cost of the inverted tree isless than the evaluation cost of the original tree. A rescheduling ofthe restructured node is not necessary in this case.

It should further be noted that if either the phrase posting listlength-based cost or the subtree cost function is used, then theoptimization stage 1020 can be performed before the scheduling stage1010.

To help further explain the phrasification, scheduling and optimizationprocesses, the following example is provided. Assume that input query is“New York restaurants OR restaurants”. The OR node with “restaurants”can be provided by the user, or by a query expansion source.

The input query has the following Boolean word tree:

(New AND York AND restaurant OR restaurants)

The query phrasification module 810 restructures the Boolean word treeas:

(new AND york AND restaurant) OR (new AND york AND restaurants)

The query phrasification module 810 generates the phrasification:

“new york restaurant”

“new york” AND “restaurant”

“new” AND “york restaurant”

“new” AND “york” AND “restaurant”

“new york restaurants”

“new york” AND “restaurants”

“new” AND “york restaurants”

“new” AND “york” AND “restaurants”

The query phrasification module 810 scores each of these phrasificationusing phrase scoring function, keeping the top N. Assume that the topselected phrases are:

“new york” AND “restaurant”

“new york” AND “restaurants”.

These are combined into a Boolean phrase tree (note that in this format,the ‘left most’ nodes are towards the bottom):

       “restaurants”     AND       “new york” OR       “restaurant”    AND       “new york”

Next, the phrase tree is scheduled by the query scheduling module 820.Assume that the phrase “new york” is assigned to index server #1 (IS1)and has length L1, “restaurant” is assigned to index server #2 (IS2) andhas length L2, and “restaurants” is assigned to index server #3 (IS3)and has length L3, where L3+L2<L1. These lengths will be used as thequery costs, and thus are assigned to the respective nodes of the phrasetree according to scheduling rule (a):

       (“restaurants”, L3, IS3)     AND        (“new york”, L1, IS1) OR       (“restaurant”, L2, IS2)     AND        (“new york”, L1, IS1)

Scheduling rule (c) is applied to the AND nodes: the children nodes aresorted by length, and the AND node is assigned to the index server ofthe leftmost child:

       (“new york”, L1, IS1)     AND: L3, IS3        (“restaurants”, L3,IS3) OR        (“new york”, L1, IS1)     AND: L2, IS2       (“restaurant”, L2, IS2)

Scheduling rule (d) is applied to the OR node. The result is:

       (“new york”, L1, IS1)     AND: L3, IS3        (“restaurants”, L3,IS3) OR: L3+L2, IS2        (“new york”, L1, IS1)     AND: L2, IS2       (“restaurant”, L2, IS2)

The OR node is assigned to its most expensive child IS2, as perscheduling rule (d).

Next, the query scheduling module 820 optimizes the schedule tree,beginning with topmost OR node and descend depth-first. The onlynontrivial operation is at this OR node, which matches node form (b), (AAND B) OR (A AND C), where A is “new york”.

In this example, the query cost is posting list length, so the phraselist length based cost function is used.

-   Applying this function is it determined that L1>L2+L3. Accordingly,    the de Morgan inversion is applied to result in the following    schedule tree:

    (“new york”, L1, IS1) AND: (L2+L3, IS3)       (“restaurants”, L3,IS3)     OR: L2+L3, IS3       (“restaurant”, L2, IS2)

In summary, once scheduling 1010 and optimization 1020 are completed,the resulting query schedule 1004 will identify for each node (being aphrase or an operator), the index server(s) 200 (or other server(s))assigned to execute the search on that node, as well as additional costinformation that can be used for further management of the search. Thequery schedule 825 is passed the query execution module 830.

4. Query Execution

The query execution module 830 of the front end server 140 isresponsible for managing the overall execution of the query schedule;this query execution module 830 is called the root query executionmodule 830. An instance of a query execution module 830 is also presentin each index server 200. Each query execution module 830 takes as inputthe query schedule 825 and provides as an output a list of documents.Each query execution module 830 is further adapted to allow thescheduled query tree to include explicit data nodes that contain acomplete posting list; this is how the results from each index server200 are passed to further index servers 200 for subsequent processing(e.g., intersection, union, negation). The explicit data nodes can alsobe used to transport additional data (e.g., scheduling information,metadata) to the index servers 200. Because each node of the queryschedule 825 includes data that identifies which index servers 200 areto execute the subtree rooted at such node, each query execution module830 can readily determine whether it is to execute whatever part of thequery schedule it receives or pass current subtree to another indexserver and process its results.

The execution process thus begins with the root query execution module830 executing on the root node of the query schedule, and proceeds byrecursive descent via the query execution modules of the index servers.Each query execution module executes the following algorithm beginningwith a root node of a received query schedule:

1) If the current node is not scheduled to execute at the currentserver:

-   -   (i) forward a subtree rooted at the current node to the        server(s) assigned to the current node. The current server is        designated a requesting server, and it receives in return one or        more document lists from the assigned server(s).    -   (ii) Return to the requesting server a document list that        results from performing the OR operation (union) on all the        document lists received from the assigned servers. In the case        of the root query execution module 830, the merged document list        serves as the search result set.

2) If the current node is scheduled at the current server, then thefollowing rules are used.

-   -   (i) If the current node is an explicit data node containing a        document list, then return the associated explicit data.    -   (ii) If the current node is a phrase node, retrieve all index        shards present on the current server that represent part of the        given phrase, and return the OR (union) of their phrase posting        lists.    -   (iii) If the current node is an OR node, execute all of the        children of this OR node (in parallel, if possible) and return        the OR (union) of their results.    -   (iv) If the current node is a NOT node, execute its child, and        return the negation of its results. This may also be done by        returning the original results together with an annotation that        the list is to be interpreted as a negation. This is done to        avoid having to return the usually large set of all results that        are not in a given set.    -   (v) If the current node is an AND node, separate the children        nodes into two groups: those children which are scheduled to        execute on the current server (local group) and those which are        scheduled to execute on other servers (remote group).        -   1) Execute all of the children in the local group, in            parallel if possible, and construct a document list (the            “local result”) which is their intersection. If there are no            children in the remote group, return the local result as the            result to the requesting server.        -   2) If the are children in the remote group, construct a new            AND node whose first child is an explicit data node            containing the local result, and whose remaining children            are the children of the remote group. Schedule this new node            using the scheduling process 1010 and then execute it,            returning the result.

The above algorithm, when completed by the various query executionmodules yields a final merged document list (at 1(ii)) resulting fromthe execution by the lower level servers on their respective portions ofthe query schedule. This document list, with additional documentinformation and annotations serves as the search result set 835.

The result set 835 is passed back to the front end server 140. Referringagain to FIG. 1, the front end server 140 ranks the documents in theresult set, using a ranking function. As noted above, the phrase postinglists can contain for each document an information retrieval score. Inone embodiment, the front end server 140 implements the second scoringalgorithm of a bifurcated document relevance scoring model, in which thefirst scoring algorithm is used during indexing to generate a phraserelevance score for a document with respect to a phrase. The result set835 includes for each document listed therein, the phrase relevancescore of the document with respect to each phrase of the query that thedocument contained. This set of phrase relevance scores for eachdocument is then input by the front end server 140 into the secondscoring algorithm, along with the query. The second scoring algorithmgenerates a final score for each document based on this input. Thedocuments are than ranked based on the final score. Additional documentinformation can be associated with each document from the documentinformation server 150. This embodiment enables the phrase posting liststo store a very small amount of data for each document. For example, aphrase posting list can store the document identifier and the phraserelevance score, and optionally flags indicating generally where in thedocument the phrase occurred (e.g., title, body, table). Thisconsiderably reduces the size of the phrase posting lists, enables moredocuments to be indexed, and reduces query access times.

The above process by which the query schedule is passed from one indexserver to another can be further improved as follows: An explicit datanode can only present in a query schedule if its parent is an AND node.This AND node will be scheduled at the same set of servers as some otherchild thereof, and the explicit data node (i.e., document list) will beintersected with this child. If this child is distributed acrossmultiple index servers—i.e., if the phrase posting list corresponding tothat child phrase is divided into multiple shards—it is only necessaryto forward to each index server that portion of the document list whichhas the potential of forming a non-empty intersection with the shard(s)served by the recipient index server. A requesting server can determinewhich documents need to be transmitted to which recipient server byusing the document number of each document and the shard assignmentfunction; this will identify which of its documents could be at therecipient server. For example, consider the query (A AND B), wherephrase A is found in a tier containing S1 shards, and phrase B is foundin a tier containing S2≧S1 shards. Then a document in shard m of A,where 0≦m<S1, is guaranteed to never intersect a document in shard n ofB, where 0≦n<S2, unless n=m (modulo S1). This immediately reduces thenumber of communications needed between servers containing shards of Aand servers containing shards of B by a factor of S2/S1.

Thus, when a document list is to be transmitted from a requesting serverto a plurality of S recipient servers corresponding to S shards of asingle posting list, the query execution module of the requesting serverprocesses the document list against the shard assignment function, anddetermine for each document therein to which recipient server thedocument should be transmitted; this will result a number of disjointsubsets of the document list, each assigned to a specific index server.For each of these subsets, the requesting query execution module createsa new query that is identical to the received query schedule but inwhich the explicit data node is replaced by the subset document list.

The query scheduling, optimization and execution process takes advantageof the tiered and shared structure of the index servers 200 to minimizethe overall execution costs for any given query. The scheduling andoptimization stages use the query cost information to efficiently assignthe index servers with respect to their tiers. The query execution stagetakes advantage of the sharding of the phrase posting lists to minimizeinter-server communications cost, both in terms of reducing the size ofthe document lists transmitted between index servers, as well aseliminating any processing that would return null results.

The example of FIGS. 12A and 12B further explains the advantages of thetiers and shards during query processing. FIG. 13A illustrates a queryschedule tree for the query (A AND B), where A is a phrase on indexserver A corresponding to Tier 0, Shard 3, and phrase B is sharded amongten index servers, index servers B₀ to B₉ corresponding to Tier 1,Shards 0 to 9. The AND node is scheduled at the same server as phrase A,that is at index server A. The pattern of execution is described below.

The root query execution module 830 initiates the execution process.Because the AND node is not assigned to the front end server 140 butinstead to index server A, the entire query schedule is transmitted toindex server A under rule (1)(i).

Index server A then executes on (A AND B). The topmost node is still theAND node, so rule (2)(v) applies. The first child, phrase A isdetermined to be a local group under rule (2)(v)(1), and so can beexecuted locally on index server A. Index server A so it does so byretrieving the phrase posting list for phrase A. The second child,phrase B, is not a local child node, but is assigned to different indexserver. Under rule (2)(v)(2), the index server A synthesizes a newquery, ([explicit data] AND B), as shown in FIG. 12B, where the explicitdata is phrase posting list for phrase A. Scheduling this new query nodeassigns the AND node to the ten index servers B₀ to B₉ containing shardsof phrase B. Index server A thus transmits this revised query schedule,rooted at the revised AND node, to these index servers.

Index servers B₀ to B₉ then execute the new AND node. Since the indexservers are tiered and sharded, it is known which documents in thephrase posting list for phrase A need to be transmitted to which indexservers B_(i): for instance, server B₀ contains only those documentswhose document identifiers are (0 mod 10), B₁ contains the documentidentifiers which are (1 mod 10), and so forth through server B9 whichcontains the document identifiers which are (9 mod 10). There may besome index servers B_(i) for which none of the document identifiersyield the index server's index number. If a document identifier which is(1 mod 10) is sent to server B₀, it will never be in the intersectionwith the documents identifiers for document in this shard of phrase B.Accordingly, index server A partitions the explicit data into no morethan ten sublists corresponding to the ten servers to which it willcommunicate and sends only those that are non-empty. If one of thosesublists is (or would be) empty, i.e. there were no hits at all in theposting list for A which were equal to (1 mod 10), then this sublist isnot sent to an index server. This demonstrates how sharding decreasesnetwork traffic.

Each index server B_(i) receives the query ([sublist data] AND B),preferably operating in parallel. Each index server B_(i) determines ifthe AND node can be executed locally. If so, the node is executed andthe index server finds only the sublist data and another node (the localshard of phrase B′s posting list) which can be executed locally. Ittherefore forms the local result, which is the intersection of thesublist and the shard of the phrase posting list, and returns it to theA server, under rule (1)(ii).

Once index server A has received the responses from all index serversB_(i), it forms the union (OR) of their results, and that is the resultof evaluating the AND. It then returns this result back to the rootquery execution module 830 of the front end server 140.

Conclusion

The present invention has been described in particular detail withrespect to one possible embodiment. Those of skill in the art willappreciate that the invention may be practiced in other embodiments.First, the particular naming of the components, capitalization of terms,the attributes, data structures, or any other programming or structuralaspect is not mandatory or significant, and the mechanisms thatimplement the invention or its features may have different names,formats, or protocols. Further, the system may be implemented via acombination of hardware and software, as described, or entirely inhardware elements. Also, the particular division of functionalitybetween the various system components described herein is merelyexemplary, and not mandatory; functions performed by a single systemcomponent may instead be performed by multiple components, and functionsperformed by multiple components may instead performed by a singlecomponent.

Some portions of above description present the features of the presentinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. These operations, while describedfunctionally or logically, are understood to be implemented by computerprograms. Furthermore, it has also proven convenient at times, to referto these arrangements of operations as modules or by functional names,without loss of generality.

Unless specifically stated otherwise as apparent from the abovediscussion, it is appreciated that throughout the description,discussions utilizing terms such as “processing” or “computing” or“calculating” or “determining” or “displaying” or the like, refer to theaction and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system memories orregisters or other such information storage, transmission or displaydevices.

Certain aspects of the present invention include process steps andinstructions described herein in the form of an algorithm. It should benoted that the process steps and instructions of the present inventioncould be embodied in software, firmware or hardware, and when embodiedin software, could be downloaded to reside on and be operated fromdifferent platforms used by real time network operating systems.

The present invention also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored on acomputer readable medium that can be accessed by the computer. Such acomputer program may be stored in a computer readable storage medium,such as, but is not limited to, any type of disk including floppy disks,optical disks, CD-ROMs, magnetic-optical disks, read-only memories(ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic oroptical cards, application specific integrated circuits (ASICs), or anytype of media suitable for storing electronic instructions, and eachcoupled to a computer system bus. Furthermore, the computers referred toin the specification may include a single processor or may bearchitectures employing multiple processor designs for increasedcomputing capability.

The algorithms and operations presented herein are not inherentlyrelated to any particular computer or other apparatus. Variousgeneral-purpose systems may also be used with programs in accordancewith the teachings herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps. The requiredstructure for a variety of these systems will be apparent to those ofskill in the, along with equivalent variations. In addition, the presentinvention is not described with reference to any particular programminglanguage. It is appreciated that a variety of programming languages maybe used to implement the teachings of the present invention as describedherein, and any references to specific languages are provided fordisclosure of enablement and best mode of the present invention.

The present invention is well suited to a wide variety of computernetwork systems over numerous topologies. Within this field, theconfiguration and management of large networks comprise storage devicesand computers that are communicatively coupled to dissimilar computersand storage devices over a network, such as the Internet.

Finally, it should be noted that the language used in the specificationhas been principally selected for readability and instructionalpurposes, and may not have been selected to delineate or circumscribethe inventive subject matter. Accordingly, the disclosure of the presentinvention is intended to be illustrative, but not limiting, of the scopeof the invention, which is set forth in the following claims.

1. A method of indexing documents based on phrases occurring in thedocuments, the method comprising: selecting a phrase posting listassociated with a phrase, and identifying a plurality of documentshaving at least one occurrence of the phrase; determining a length ofthe phrase posting list; responsive to the length of the phrase postinglist being less than a first predetermined length, associating thephrase posting list with one of a plurality of first tier index servers;responsive to the length of the phrase posting list being greater thanthe first predetermined length: dividing the phrase posting list into aplurality of shards, each shard including a subset of the plurality ofthe documents; and associating each phrase posting list shard with acorresponding selected second tier index server, wherein the number ofshards correspond to the number of second tier index servers.